'\"macro stdmacro
.\"
.\" Copyright (c) 2018-2020 Red Hat.
.\"
.\" This program is free software; you can redistribute it and/or modify it
.\" under the terms of the GNU General Public License as published by the
.\" Free Software Foundation; either version 2 of the License, or (at your
.\" option) any later version.
.\"
.\" This program is distributed in the hope that it will be useful, but
.\" WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
.\" or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
.\" for more details.
.\"
.\"
.TH PMSERIES 1 "PCP" "Performance Co-Pilot"
.SH NAME
\f3pmseries\f1 \- display information about performance metric timeseries
.SH SYNOPSIS
\fBpmseries\fR
[\fB\-adFiIlLmMnqsStvV?\fR]
[\fB\-c\fR \fIconfig\fR]
[\fB\-g\fR \fIpattern\fR]
[\fB\-h\fR \fIhost\fR]
[\fB\-p\fR \fIport\fR]
[\fB\-Z\fR \fItimezone\fR]
[\fIquery\fR | \fIlabels\fR ... | \fIseries\fR ... | \fIsource\fR ... ]
.SH DESCRIPTION
.de SAMPLE
.RS 2n
.nf
.nh
..
.de ESAMPLE
.hy
.fi
.RE
..
.B pmseries
displays various types of information about performance metrics
available through the scalable timeseries facilities of the
Performance Co-Pilot (PCP) and the Redis distributed data store.
.PP
By default
.B pmseries
communicates with a local
.BR redis-server (1),
however the \fB\-h\fR and \fB\-p\fR options can be used to
specify an alternate Redis instance.
If this instance is a node of a Redis cluster, all
other instances in the cluster will be discovered
and used automatically.
.PP
.B pmseries
runs in several different modes \- either querying
timeseries identifiers, metadata or values (already
stored in Redis), or manually loading timeseries
into Redis.
The latter mode is seldom used, however, since
.BR pmproxy (1)
will automatically perform this function for local
.BR pmlogger (1)
instances,
when running in its default time series mode.
.PP
Without command line options specifying otherwise,
.B pmseries
will issue a timeseries query to find matching timeseries and
values.
All timeseries are identified using a unique SHA-1 hash which
is always displayed in a 40-hexdigit human readable form.
These hashes are formed using the metadata associated with
every metric.
.PP
Importantly, this includes all metric metadata (labels, names,
descriptors).
Metric labels in particular are (as far as possible) unique for
every machine \- on Linux for example the labels associated
with every metric include the unique \fI/etc/machine-id\fR,
the hostname, domainname, and other automatically generated
machine labels, as well as any administrator-defined labels
from \fI/etc/pcp/labels\fR.
These labels can be reported with
.BR pminfo (1)
and the \fIpmcd.labels\fR metric.
.PP
See
.BR pmLookupLabels (3),
.BR pmLookupInDom (3),
.BR pmLookupName (3)
and
.BR pmLookupDesc (3)
for detailed information about metric labels and other metric
metadata used in each timeseries identifier hash calculation.
.PP
The timeseries identifiers provide a higher level (and machine
independent) identifier than the traditional PCP performance
metric identifiers (pmID), instance domain identifiers (pmInDom)
and metric names.
See
.BR PCPIntro (1)
for more details about these traditional identifiers.
However,
.B pmseries
uses timeseries identifiers in much the same way that
.BR pminfo (1)
uses the lower level indom, metric identifiers and metric names.
.PP
The default mode of
.B pmseries
operation (i.e. with no command line options) depends on the
arguments it is presented.
If all non-option arguments appear to be timeseries
identifiers (in 40 hex digit form)
.B pmseries
will report metadata for these timeseries \- refer to
the \fB\-a\fR option for details.
Otherwise, the parameters will be treated as a timeseries
query.
.SH TIMESERIES QUERIES
Query expressions are formed using the
.B pmseries
query language described below, but can be as simple as a
metric name.
.P
The following is an example of querying timeseries from all
hosts that match a metric name pattern (globbed):
.P
.SAMPLE
$ pmseries kernel.all.cpu*
1d7b0bb3f6aec0f49c54f5210885464a53629b60
379db729afd63fb9eff436625bd6c55a7adc5cfd
3dd3b45bb05f96636043e5d58b52b441ce542285
[...]
ed2bf325ff6dc7589ec966698e5404b67252306a
dcb2a032a308b5717bf605ba8f8737e9c6e1ed19
.ESAMPLE
.PP
To identify timeseries expression operands, the query language uses the general
syntax:
.PP
.SAMPLE
[\fImetric.name\fR] '\fB{\fImetadata qualifiers\fB}\fR' '\fB[\fItime specification\fB]\fR'
.ESAMPLE
.PP
The \fImetric.name\fR component restricts the timeseries query
to any matching PCP metric name (the list of metric names for a
PCP archive or live host is reported by
.BR pminfo (1)
with no arguments beyond \-\-\fBhost\fR or \-\-\fBarchive\fR).
The
.B pmseries
syntax extends on that of
.B pminfo
and allows for
.BR glob (7)
based pattern matching within the metric name.
The above describes operands available as the leaves of
.B pmseries
expressions,
which may include functions, arithmetic operators and other features.
See the
.B EXPRESSIONS
section below for further details.
.SH METADATA QUALIFIERS AND METADATA OPERATORS
Metadata qualifiers are enclosed by ``curly'' braces (\fB{}\fR),
and further restrict the query results to timeseries operands with various
metadata properties.
These qualifiers are based on metric or instance names, and
metric label values, and take the general form
\fImetadata.name\fR OPERATOR \fIvalue\fR, such as:
.P
.SAMPLE
instance.name == "cpu0"
metric.name != "kernel.all.pswitch"
.ESAMPLE
.PP
When using label names, the metadata qualifier is optional and
can be dropped, such as:
.P
.SAMPLE
label.hostname == "www.acme.com"
hostname == "www.acme.com"
.ESAMPLE
.PP
For metric and instance names only the string operators apply,
but for metric label values all operators are available.
The set of available operators is:
.SS Boolean operators
All string (label, metrics and instances) and numeric
(label) values can be tested for equality ("==") or
inequality ("!=").
.SS String operators
Strings can be subject to pattern matching in the form of
glob matching ("~~"), regular expression matching ("=~"),
and regular expression non-matching ("!~").
The ":" operator is equivalent to "~~" - i.e., regular expression
matching.
.SS Relational operators (numeric label values only)
Numeric label values can be subject to the less than ("<"),
greater than (">"), less than or equal ("<="), greater than
or equal (">="), equal ("==") and not equal ("!=") operators.
.SS Logical operators
Multiple metadata qualifiers can be combined with the logical
operators for AND ("&&") and OR ("||") as in many programming
languages.
The comma (",") character is equivalent to logical AND ("&&").
.SH TIME SPECIFICATION
The final (optional) component of a query allows the user to
specify a specific time window of interest.
Any time specification will result in values being returned
for all matching timeseries only for the time window specified.
.PP
The specification is ``square'' bracket (\fB[]\fR) enclosed,
and consists of one or more comma-separated components.
Each component specifies some aspect related to time, taking
the general form: \fBkeyword\fR: \fIvalue\fR, such as:
.P
.SAMPLE
samples:10
.ESAMPLE
.SS Sample count
The number of samples to return, specified via either the
.B samples
or (equivalent)
.B count
keyword.
The
.I value
provided must be a positive integer.
If no end time is explicitly set (see ``Time window'' later)
then the most recent samples will be returned.
.SS Sample interval
An interval between successive samples can be requested using
the
.B interval
or (equivalent)
.B delta
keyword.
The
.I value
provided should be either a numeric or string value that will
be parsed by
.BR pmParseInterval (3),
such as \fB5\fR (seconds) or \fB2min\fR (minutes).
.SS Time window
Start and end times, and alignments, affecting the returned
values.
The keywords match the parameters to the
.BR pmParseTimeWindow (3)
function which will be used to parse them, and are:
.B start
or (equivalent)
.BR begin ,
.B finish
or (equivalent)
.BR end ,
.B align
and
.BR offset .
.SS Time zones
The resulting timestamps can be returned having been evaluated
for a specific timezone, using the
.B timezone
or
.B hostzone
keywords.
The
.I value
associated with
.B timezone
will be interpreted by
.BR pmNewZone (3).
A
.B true
or
.B false
value should be associated with
.BR hostzone ,
and when set to
.B true
this has the same effect as described by
.BR pmNewContextZone (3).
.SH EXPRESSIONS
As described above, operands are the leaves of a query expression tree.
.P
.SAMPLE
[\fImetric.name\fR] '\fB{\fImetadata qualifiers\fB}\fR' '\fB[\fItime specification\fB]\fR'
.ESAMPLE
Note in most of the query expression examples below, the \fImetadata qualifiers\fP
have been omitted for brevity.
In all cases, multiple time series may qualify, particularly for the \fBhostname\fP label.
.PP
In the simple case, a query expression consists of a single operand
and may just be a metric name.
In the more general case, a query expression is either an operand
or the argument to a function, or two operands in a binary arithmetic or logical expression.
Most functions take a single argument (an expression), though some require additional
arguments, e.g.
.BR rescale .
.P
.SAMPLE
\fIoperand\fP | \fIexpr\fP \fIoperator\fP \fIexpr\fP | \fIfunc\fP(\fIexpr\fP[, \fIarg\fP])
.ESAMPLE
.P
This grammar shows expressions may be nested, e.g. using the addition (\fB+\fP) operator as an example,
.P
.SAMPLE
\fIfunc1\fP\fB(\fP\fIfunc2\fP\fB(\fP\fIexpr\fP\fB))\fP
\fIfunc1\fP\fB(\fP\fIexpr\fP\fB)\fP \fB+\fP \fIfunc2\fP\fB(\fP\fIexpr\fP\fB)\fP
\fIexpr\fP \fB+\fP \fIfunc\fP\fB(\fP\fIexpr\fP\fB)\fP
\fIfunc\fP\fB(\fP\fIexpr\fP\fB)\fP \fB+\fP \fIexpr\fP
\fIexpr\fP \fB+\fP \fIexpr\fP
.ESAMPLE
.PP
Rules governing compatibility of operands in an expression generally depend on
the function and/or operators and are described below individually.
An important rule is that if any time windows are specified, then
all operands must cover the same number of samples,
though the time windows may differ individually.
If no time windows or sample counts are given, then
.B pmseries
will return a series identifier (SID) instead of a series of timestamps and values.
This SID may be used in subsequent
.B /series/values?series=\fISID\fP
RESTAPI calls, along with a specific time window.
.SS Arithmetic Operators
.B pmseries
support addition, subtraction, division and multiplication on each value in the
time series of a binary pair of operands.
No unary or ternary operators are supported (yet).
In all cases, the instance domain and the number of samples of
time series operands must be the same.
The metadata (units and dimensions) must also be compatible.
Depending on the function, the result will usually have the same
instance domain and (unless noted otherwise), the same units as
the operands.
The metadata dimensions (space, time, count) of the result may
differ (see below).
.PP
Expression operands may have different qualifiers, e.g. you can perform binary
arithmetic on metrics qualified by different labels (such as \fBhostname\fP),
or metric names.
For example, to add the two most recent samples of the process context
switch (pswitch) counter metric for hosts
.B node88
and
.BR node89 ,
and then perform rate conversion:
.P
.SAMPLE
$ pmseries 'rate(kernel.all.pswitch{hostname:\fBnode88\fP}[count:2] +
                 kernel.all.pswitch{hostname:\fBnode89\fP}[count:2])'
1cf1a85d5978640ef94c68264d3ae8866cc11f7c
    [Tue Nov 10 14:39:48.771868000 2020] 71.257509 8e0a59304eb99237b89593a3e839b5bb8b9a9924
.ESAMPLE
.P
Note the resulting time series of values has one less sample than the expression
operand passed to the
.B rate
function.
.PP
Other rules for arithmetic expressions:
.TP
1. if both operands have the semantics of a counter, then only addition and subtraction are allowed
.TP
2. if the left operand is a counter and the right operand is not, then only multiplication or division are allowed
.TP
3. if the left operand is not a counter and the right operand is a counter, then only multiplication is allowed.
.TP
4. addition and subtraction - the dimensions of the result are the same as the dimensions of the operands.
.TP
5. multiplication - the dimensions of the result are the sum of the dimensions of the operands.
.TP
6. division - the dimensions of the result are the difference of the dimensions of the operands.
.SS Functions
Expression functions operate on vectors of time series values,
and may be nested with other functions or expressions as described above.
When an operand has multiple instances, there will generally be one result
for each series of instances.
For example, the result for
.P
.SAMPLE
$ pmseries 'min(kernel.all.load[count:100])'
.ESAMPLE
.P
will be the smallest value of the 100 most recent samples,
treating each of the three load average instances as a separate time series.
As an example, for the two most recent samples for each of the
three instances of the load average metric:
.P
.SAMPLE
$ pmseries 'kernel.all.load[count:2]'
726a325c4c1ba4339ecffcdebd240f441ea77848
    [Tue Nov 10 11:52:30.833379000 2020] 1.100000e+00 a7c96e5e2e0431a12279756d11590fa9fed8f306
    [Tue Nov 10 11:52:30.833379000 2020] 9.900000e-01 ee9b506935fd0976a893dc27242926f49326b9a1
    [Tue Nov 10 11:52:30.833379000 2020] 1.070000e+00 d5e1c360d13064c461169091997e1e8be7488133
    [Tue Nov 10 11:52:20.827134000 2020] 1.120000e+00 a7c96e5e2e0431a12279756d11590fa9fed8f306
    [Tue Nov 10 11:52:20.827134000 2020] 9.900000e-01 ee9b506935fd0976a893dc27242926f49326b9a1
    [Tue Nov 10 11:52:20.827134000 2020] 1.070000e+00 d5e1c360d13064c461169091997e1e8be7488133
.ESAMPLE
.P
Using the \fBmin\fP function :
.P
.SAMPLE
$ pmseries 'min(kernel.all.load[count:2])'
11b965bc5f9598034ed9139fb3a78c6c0b7065ba
    [Tue Nov 10 11:52:30.833379000 2020] 1.100000e+00 a7c96e5e2e0431a12279756d11590fa9fed8f306
    [Tue Nov 10 11:52:30.833379000 2020] 9.900000e-01 ee9b506935fd0976a893dc27242926f49326b9a1
    [Tue Nov 10 11:52:30.833379000 2020] 1.070000e+00 d5e1c360d13064c461169091997e1e8be7488133
.ESAMPLE
.P
For singular metrics (with no instance domain), a single value will result,
e.g. for the five most recent samples of the context switching metric:
.P
.SAMPLE
$ pmseries 'kernel.all.pswitch[count:5]'
d7832c4fba33bcc980b1a1b614e0508043288480
    [Tue Nov 10 12:44:59.380666000 2020] 460774294
    [Tue Nov 10 12:44:49.382070000 2020] 460747232
    [Tue Nov 10 12:44:39.378545000 2020] 460722370
    [Tue Nov 10 12:44:29.379029000 2020] 460697388
    [Tue Nov 10 12:44:19.379096000 2020] 460657412

$ pmseries 'min(kernel.all.pswitch[count:5])'
1b6e92fb5bc012372f54452734dd03f0f131fa06
    [Tue Nov 10 12:44:19.379096000 2020] 460657412 d7832c4fba33bcc980b1a1b614e0508043288480

.ESAMPLE
.P
Future versions of
.B pmseries
may provide functions that perform aggregation, interpolation, filtering
or transforms in other ways, e.g. across instances instead of time.
.SS Function Reference
.BR max(\f2expr\fP)
the maximum value in the time series for each instance of \fIexpr\fP
.P
.BR min(\f2expr\fP)
the minimum value in the time series for each instance of \fIexpr\fP
.P
.BR rate(\f2expr\fP)
the rate with respect to time of each sample.
The given \fIexpr\fP must have
.B counter
semantics
and the result will have
.B instant
semantics (the time dimension reduced by one).
In addition, the result will have one less sample than the operand - this
is because the first sample cannot be rate converted (two samples are required).
.P
.BR rescale(\f2expr\fP, \f2scale\fP)
rescale the values in the time series for each instance of
.I expr
to
.I scale
(units).
Note that
.I expr
should have
.B instant
or
.B discrete
semantics (not
.B counter
- rate conversion should be done first if needed).
The time, space and count dimensions between
.I expr
and
.I scale
must be compatible.
Example:
rate convert the read throughput counter for each disk instance
and then rescale to mbytes per second.
Note the native units of
.B disk.dev.read_bytes
is a
.B counter
of kbytes read from each device instance since boot.
.P
.SAMPLE
$ pmseries 'rescale(rate(disk.dev.read_bytes[count:4]), "mbytes/s")'
.ESAMPLE
.P
.BR abs(\f2expr\fP)
the absolute value of each value in the time series for each instance
of \fIexpr\fP.
This has no effect if the type of \fIexpr\fP is unsigned.
.P
.BR floor(\f2expr\fP)
rounded down to the nearest integer value of the time series for each
instance of \fIexpr\fP.
.P
.BR round(\f2expr\fP)
rounded up or down to the nearest integer for each value in the time series
for each instance of \fIexpr\fP.
.P
.BR log(\f2expr\fP)
logarithm of the values in the time series for each instance of \fIexpr\fP
.P
.BR sqrt(\f2expr\fP)
square root of the values in the time series for each instance of \fIexpr\fP
.SS Compatibility
All operands in an expression must have the same number of samples,
but not necessarily the same time window. e.g. you could subtract some
metric time series from today from that of yesterday by giving different
time windows and different metrics or qualifiers, ensuring the same number
of samples are given as the operands.
.PP
Operands in an expression must either all have a time window, or none.
If no operands have a time window, then instead of a series of time stamps
and values, the result will be a time series identifier (\f2SID\fP)
that may be passed to the
.B /series/values?series=\f2SID\fP
REST API function, along with a time window.
For further details, see
.BR PMWEBAPI (3).
.PP
If the semantics of both operands in an arithmetic expression
are not counter (i.e. \fBPM_SEM_INSTANT\fP or \fBPM_SEM_DISCRETE\fP) then the
result will have semantics \fBPM_SEM_INSTANT\fP unless both
operands are \fBPM_SEM_DISCRETE\fP in which case the result
is also \fBPM_SEM_DISCRETE\fP.
.SH TIMESERIES METADATA
Using command line options,
.B pmseries
can be requested to provide metadata (metric names, instance
names, labels, descriptors) associated with either individual
timeseries or a group of timeseries, for example:
.PP
.SAMPLE
$ pmseries -a dcb2a032a308b5717bf605ba8f8737e9c6e1ed19

dcb2a032a308b5717bf605ba8f8737e9c6e1ed19
    PMID: 60.0.21
    Data Type: 64-bit unsigned int  InDom: PM_INDOM_NULL 0xffffffff
    Semantics: counter  Units: millisec
    Source: f5ca7481da8c038325d15612bb1c6473ce1ef16f
    Metric: kernel.all.cpu.nice
    labels {"agent":"linux","domainname":"localdomain",\\
            "groupid":1000,"hostname":"shard",\\
            "latitude":-25.28496,"longitude":152.87886,\\
            "machineid":"295b16e3b6074cc8bdbda8bf96f6930a",\\
            "userid":1000}
.ESAMPLE
.PP
The complete set of
.B pmseries
metadata reporting options are:
.TP 5
\fB\-a\fR, \fB\-\-all\fR
Convenience option to report all metadata for the given timeseries,
equivalent to \fB\-deilms\fR.
.TP
\fB\-d\fR, \fB\-\-desc\fR
Metric descriptions detailing the PMID, data type, data semantics, units,
scale and associated instance domain.
This option has a direct \fBpminfo\fR(1) equivalent.
.TP
\fB\-g\fR \fIpattern\fR, \fB\-\-glob\fR=\fIpattern\fR
Provide a
.BR glob (7)
.I pattern
to restrict the report provided by the \fB\-i\fR, \fB\-l\fR, \fB\-m\fR,
and \fB\-S\fR.
.TP
\fB\-i\fR, \fB\-\-instances\fR
Metric descriptions detailing the PMID, data type, data semantics, units,
scale and associated instance domain.
.TP
\fB\-I\fR, \fB\-\-fullindom\fR
Print the InDom in verbose mode.
This option has a direct \fBpminfo\fR(1) equivalent.
.TP
\fB\-l\fR, \fB\-\-labels\fR
Print label sets associated with metrics and instances.
Labels are optional metric metadata described in detail in
.BR pmLookupLabels (3).
This option has a direct \fBpminfo\fR(1) equivalent.
.TP
\fB\-m\fR, \fB\-\-metrics\fR
Print metric names.
.TP
\fB\-M\fR, \fB\-\-fullpmid\fR
Print the PMID in verbose mode.
This option has a direct \fBpminfo\fR(1) equivalent.
.TP
\fB\-n\fR, \fB\-\-names\fR
Print comma-separated label names only (not values) for the labels
associated with metrics and instances.
.TP
\fB\-s\fR, \fB\-\-series\fR
Print timeseries identifiers associated with metrics, instances and
sources.
These unique identifiers are calculated from intrinsic (non-optional)
labels and other metric metadata associated with each PMAPI context
(sources), metrics and instances.
Archive, local context or
.BR pmcd (1)
connections for the same host all produce the same source identifier.
This option has a direct \fBpminfo\fR(1) equivalent.
See also
.BR pmLookupLabels (3)
and the \fB\-l/\fB\-\-labels\fR option.
.SH TIMESERIES SOURCES
A source is a unique identifier (represented externally as a 40-byte
hexadecimal SHA-1 hash) that represents both the live host and/or
archives from which each timeseries originated.
The context for a source identifier (obtained with \fB\-s\fR) can be
reported with:
.TP 5
\fB\-S\fR, \fB\-\-sources\fR
Print names for timeseries sources.
These names are either hostnames or fully qualified archive paths.
.PP
It is important to note that live and archived sources can and will
generate the same SHA-1 source identifier hash, provided that
the context labels remain the same for that host (labels are stored
in PCP archives and can also be fetched live from
.BR pmcd (1)).
.SH TIMESERIES LOADING
Timeseries metadata and data are loaded either automatically
by a local
.BR pmproxy (1),
or manually using a specially crafted
.B pmseries
query and the \fB-L\fR/\fB\-\-load\fR option:
.PP
.SAMPLE
$ pmseries --load "{source.path: \\"$PCP_LOG_DIR/pmlogger/acme\\"}"
pmseries: [Info] processed 2275 archive records from [...]
.ESAMPLE
.PP
This query must specify a source archive path, but can also restrict
the import to specific timeseries (using metric names, labels, etc)
and to a specific time window using the time specification component
of the query language.
.PP
As a convenience, if the argument to load is a valid file path as
determined by
.BR access (2),
then a short-hand form can be used:
.PP
.SAMPLE
$ pmseries --load $PCP_LOG_DIR/pmlogger/acme.0
.ESAMPLE
.SH OPTIONS
The available command line options, in addition to timeseries
metadata and sources options described above, are:
.TP 5
\fB\-c\fR \fIconfig\fR, \fB\-\-config\fR=\fIconfig\fR
Specify the
.IR config
file to use.
.TP
\fB\-h\fR \fIhost\fR, \fB\-\-host\fR=\fIhost\fR
Connect Redis server at
.IR host ,
rather than the one the localhost.
.TP
\fB\-L\fR, \fB\-\-load\fR
Load timeseries metadata and data into the Redis cluster.
.TP
\fB\-p\fR \fIport\fR, \fB\-\-port\fR=\fIport\fR
Connect Redis server at
.IR port ,
rather than the default \fB6379\fR.
.TP
\fB\-q\fR, \fB\-\-query\fR
Perform a timeseries query.
This is the default action.
.TP
\fB\-t\fR, \fB\-\-times\fR
Report time stamps numerically (in milliseconds) instead of
the default human readable form.
.TP
\fB\-v\fR, \fB\-\-values\fR
Report all of the known values for given \fIlabel\fR name(s).
.TP
\fB\-V\fR, \fB\-\-version\fR
Display version number and exit.
.TP
\fB\-Z\fR \fItimezone\fR, \fB\-\-timezone\fR=\fItimezone\fR
Use
.I timezone
for the date and time.
.I Timezone
is in the format of the environment variable
.B TZ
as described in
.BR environ (7).
.TP
\fB\-?\fR, \fB\-\-help\fR
Display usage message and exit.
.SH EXAMPLES
The following sample query shows several fundamental aspects of the
.B pmseries
query language:
.PP
.SAMPLE
$ pmseries 'kernel.all.load{hostname:"toium"}[count:2]'

eb713a9cf472f775aa59ae90c43cd7f960f7870f
    [Thu Nov 14 05:57:06.082861000 2019] 1.0e-01 b84040ffccd54f839b65140cf139bab51cbbcf62
    [Thu Nov 14 05:57:06.082861000 2019] 6.8e-01 a60b5b3bf25e71071c41934fa4d7d251f765f30c
    [Thu Nov 14 05:57:06.082861000 2019] 6.4e-01 e1974a062375e6e62370ffadf5b0650dad739480
    [Thu Nov 14 05:57:16.091546000 2019] 1.6e-01 b84040ffccd54f839b65140cf139bab51cbbcf62
    [Thu Nov 14 05:57:16.091546000 2019] 6.7e-01 a60b5b3bf25e71071c41934fa4d7d251f765f30c
    [Thu Nov 14 05:57:16.091546000 2019] 6.4e-01 e1974a062375e6e62370ffadf5b0650dad739480
.ESAMPLE
.PP
This query returns the two most recent values for all instances of
the
.B kernel.all.load
metric with a
.I label.hostname
matching the regular expression "toium".
This is a set-valued metric (i.e., a metric with an ``instance
domain'' which in this case consists of three instances: 1, 5
and 15 minute averages).
The first column returned is a timestamp, then a floating point
value, and finally an instance identifier timeseries hash (two
values returned for three instances, so six rows are returned).
The metadata for these timeseries can then be further examined:
.PP
.SAMPLE
$ pmseries -a eb713a9cf472f775aa59ae90c43cd7f960f7870f

eb713a9cf472f775aa59ae90c43cd7f960f7870f
    PMID: 60.2.0
    Data Type: float  InDom: 60.2 0xf000002
    Semantics: instant  Units: none
    Source: 0e89c1192db79326900d82131c31399524f0b3ee
    Metric: kernel.all.load
    inst [1 or "1 minute"] series b84040ffccd54f839b65140cf139bab51cbbcf62
    inst [5 or "5 minute"] series a60b5b3bf25e71071c41934fa4d7d251f765f30c
    inst [15 or "15 minute"] series e1974a062375e6e62370ffadf5b0650dad739480
    inst [1 or "1 minute"] labels {"agent":"linux","hostname":"toium"}
    inst [5 or "5 minute"] labels {"agent":"linux","hostname":"toium"}
    inst [15 or "15 minute"] labels {"agent":"linux","hostname":"toium"}
.ESAMPLE
.SH PCP ENVIRONMENT
Environment variables with the prefix \fBPCP_\fP are used to parameterize
the file and directory names used by PCP.
On each installation, the
file \fI/etc/pcp.conf\fP contains the local values for these variables.
The \fB$PCP_CONF\fP variable may be used to specify an alternative
configuration file, as described in \fBpcp.conf\fP(5).
.PP
For environment variables affecting PCP tools, see \fBpmGetOptions\fP(3).
.SH SEE ALSO
.BR PCPIntro (1),
.BR pmcd (1),
.BR pminfo (1),
.BR pmproxy (1),
.BR redis-server (1),
.BR access (2),
.BR PMAPI (3),
.BR PMWEBAPI (3),
.BR pmLookupDesc (3),
.BR pmLookupInDom (3),
.BR pmLookupLabels (3),
.BR pmLookupName (3),
.BR pmNewContextZone (3),
.BR pmNewZone (3),
.BR pmParseInterval (3),
.BR pmParseTimeWindow (3),
.BR pcp.conf (5),
.BR environ (7),
.BR glob (7)
and
.BR regex (7).
